Graphical user interface for displaying guidance information during an image-guided procedure

ABSTRACT

A method for displaying guidance information during an image-guided surgical procedure comprises receiving, by one or more hardware processors, data from a tracking system associated with an elongate device comprising a flexible body and calculating, by the one or more hardware processors, at least one condition along a length of the flexible body based on the data. The method further comprises determining, by the one or more hardware processors, supplemental guidance information based on the at least one condition and augmenting, by the one or more hardware processors, one or more images with the supplemental guidance information to produce one or more augmented images. The method further comprises displaying the one or more augmented images on a display device at a surgeon console.

RELATED APPLICATIONS

This patent application claims priority to and the benefit of the filingdate of U.S. Provisional Patent Application No. 62/486,879 entitled“Graphical User Interface for Monitoring an Image-Guided Procedure,”filed Apr. 18, 2017 and to U.S. Provisional Patent Application No.62/357,217 entitled “Graphical User Interface for Displaying GuidanceInformation During an Image-Guided Procedure” filed Jun. 30, 2016 whichis incorporated by reference herein in its entirety. The presentdisclosure is related to U.S. Provisional Patent Application 62/357,258,entitled “Graphical User Interface for Displaying Guidance Informationin a Plurality of Modes During an Image-Guided Procedure,” filed Jun.30, 2016; U.S. Provisional Patent Application 62/357,272, entitled“Systems and Methods of Steerable Elongate Device,” filed Jun. 30, 2016;and PCT/US2017/039808, entitled “Systems and Methods of SteerableElongate Device,” filed Jun. 28, 2017, which are hereby incorporated byreference in their entirety.

TECHNICAL FIELD

The present disclosure is directed to systems and methods for conductingan image-guided procedure and more particularly to systems and methodsfor displaying guidance information during an image-guided procedure.

BACKGROUND

Minimally invasive medical techniques are intended to reduce the amountof tissue that is damaged during medical procedures, thereby reducingpatient recovery time, discomfort, and harmful side effects. Suchminimally invasive techniques may be performed through natural orificesin a patient anatomy or through one or more surgical incisions. Throughthese natural orifices or incisions clinicians may insert minimallyinvasive medical instruments (including surgical, diagnostic,therapeutic, or biopsy instruments) to reach a target tissue location.One such minimally invasive technique is to use a flexible and/orsteerable elongate device, such as a catheter, that can be inserted intoanatomic passageways and navigated toward a region of interest withinthe patient anatomy. Control of such an elongate device by medicalpersonnel during an image-guided procedure involves the management ofseveral degrees of freedom including at least the management ofinsertion and retraction of the elongate device as well as steeringand/or bend radius of the device. In addition, different modes ofoperation may also be supported.

Accordingly, it would be advantageous to provide a graphical userinterface that supports intuitive control and management of flexibleand/or steerable elongate devices, such as steerable catheters, that aresuitable for use during minimally invasive medical techniques.

SUMMARY

The embodiments of the invention are best summarized by the claims thatfollow the description.

A method for displaying guidance information during an image-guidedsurgical procedure comprises receiving, by one or more hardwareprocessors, data from a tracking system associated with an elongatedevice comprising a flexible body and calculating, by the one or morehardware processors, at least one condition along a length of theflexible body based on the data. The method further comprisesdetermining, by the one or more hardware processors, supplementalguidance information based on the at least one condition and augmenting,by the one or more hardware processors, one or more images with thesupplemental guidance information to produce one or more augmentedimages. The method further comprises displaying the one or moreaugmented images on a display device at a surgeon console. Anon-transitory machine-readable medium comprises a plurality ofmachine-readable instructions which when executed by one or moreprocessors associated with the medical device are adapted to cause theone or more processors to perform the method for displaying guidanceinformation.

A medical device comprising an elongate device including a flexible bodyand a tracking system disposed along at least a portion of the flexiblebody. The medical system also comprises one or more processors coupledto the tracking system. The one or more processors are configured toreceive data from the tracking system, calculate at least one conditionalong a length of the flexible body based on the received data, anddetermine supplemental guidance information based on the at least onecondition. The one or more processors are further configured to augmentone or more images using the supplemental guidance information toproduce one or more augmented images and display the one or moreaugmented images.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory innature and are intended to provide an understanding of the presentdisclosure without limiting the scope of the present disclosure. In thatregard, additional aspects, features, and advantages of the presentdisclosure will be apparent to one skilled in the art from the followingdetailed description.

BRIEF DESCRIPTIONS OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 is a simplified diagram of a teleoperated medical systemaccording to some embodiments.

FIG. 2A is a simplified diagram of a medical instrument system accordingto some embodiments.

FIG. 2B is a simplified diagram of a medical instrument with an extendedmedical tool according to some embodiments.

FIGS. 3A and 3B are simplified diagrams of side views of a patientcoordinate space including a medical instrument mounted on an insertionassembly according to some embodiments.

FIG. 4 is a simplified diagram of a graphical user interface fordisplaying supplemental guidance information for use in an image-guidedsurgical procedure according to some embodiments.

FIG. 5 is a simplified diagram of a window for displaying imagesaugmented with supplemental guidance information according to someembodiments.

FIG. 6 is a simplified diagram of an actuation information window thatdisplays actuation information including supplemental guidanceinformation according to some embodiments.

FIG. 7 is a simplified diagram of a method of displaying supplementalguidance information during an image-guided surgical procedure accordingto some embodiments.

FIG. 8 is a screenshot of a graphical user interface displayingsupplemental guidance information for use in an image-guided surgicalprocedure according to some embodiments.

FIG. 9 is a simplified diagram of a bend indicator according to someembodiments.

Embodiments of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures, whereinshowings therein are for purposes of illustrating embodiments of thepresent disclosure and not for purposes of limiting the same.

DETAILED DESCRIPTION

In the following description, specific details are set forth describingsome embodiments consistent with the present disclosure. Numerousspecific details are set forth in order to provide a thoroughunderstanding of the embodiments. It will be apparent, however, to oneskilled in the art that some embodiments may be practiced without someor all of these specific details. The specific embodiments disclosedherein are meant to be illustrative but not limiting. One skilled in theart may realize other elements that, although not specifically describedhere, are within the scope and the spirit of this disclosure. Inaddition, to avoid unnecessary repetition, one or more features shownand described in association with one embodiment may be incorporatedinto other embodiments unless specifically described otherwise or if theone or more features would make an embodiment non-functional.

In some instances well known methods, procedures, components, andcircuits have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

This disclosure describes various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian x-, y-, and z-coordinates). Asused herein, the term “orientation” refers to the rotational placementof an object or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an object.

FIG. 1 is a simplified diagram of a teleoperated medical system 100(also called “teleoperational manipulator assembly”) according to someembodiments. In some embodiments, teleoperated medical system 100 may besuitable for use in, for example, surgical, diagnostic, therapeutic, orbiopsy procedures. As shown in FIG. 1 , medical system 100 generallyincludes a teleoperational manipulator assembly 102 for operating amedical instrument 104 in performing various procedures on a patient P.Teleoperational manipulator assembly 102 is mounted to or near anoperating table T. A master assembly 106 allows an operator (e.g., asurgeon, a clinician, or a physician O as illustrated in FIG. 1 ) toview the interventional site and to control teleoperational manipulatorassembly 102.

Master assembly 106 may be located at a surgeon's console which isusually located in the same room as operating table T, such as at theside of a surgical table on which patient P is located. However, itshould be understood that physician O can be located in a different roomor a completely different building from patient P. Master assembly 106generally includes one or more control devices for controllingteleoperational manipulator assembly 102. The control devices mayinclude any number of a variety of input devices, such as joysticks,trackballs, data gloves, trigger-guns, hand-operated controllers, voicerecognition devices, body motion or presence sensors, and/or the like.To provide physician O a strong sense of directly controllinginstruments 104 the control devices may be provided with the samedegrees of freedom as the associated medical instrument 104. In thismanner, the control devices provide physician O with telepresence or theperception that the control devices are integral with medicalinstruments 104.

In some embodiments, the control devices may have more or fewer degreesof freedom than the associated medical instrument 104 and still providephysician O with telepresence. In some embodiments, the control devicesmay optionally be manual input devices which move with six degrees offreedom, and which may also include an actuatable handle for actuatinginstruments (for example, for closing grasping jaws, applying anelectrical potential to an electrode, delivering a medicinal treatment,and/or the like).

Teleoperational manipulator assembly 102 supports medical instrument 104and may include a kinematic structure of one or more non-servocontrolled links (e.g., one or more links that may be manuallypositioned and locked in place, generally referred to as a set-upstructure) and a teleoperational manipulator. Teleoperationalmanipulator assembly 102 may optionally include a plurality of actuatorsor motors that drive inputs on medical instrument 104 in response tocommands from the control system (e.g., a control system 112). Theactuators may optionally include drive systems that when coupled tomedical instrument 104 may advance medical instrument 104 into anaturally or surgically created anatomic orifice. Other drive systemsmay move the distal end of medical instrument 104 in multiple degrees offreedom, which may include three degrees of linear motion (e.g., linearmotion along the X, Y, Z Cartesian axes) and in three degrees ofrotational motion (e.g., rotation about the X, Y, Z Cartesian axes).Additionally, the actuators can be used to actuate an articulable endeffector of medical instrument 104 for grasping tissue in the jaws of abiopsy device and/or the like. Actuator position sensors such asresolvers, encoders, potentiometers, and other mechanisms may providesensor data to medical system 100 describing the rotation andorientation of the motor shafts. This position sensor data may be usedto determine motion of the objects manipulated by the actuators.

Teleoperated medical system 100 may include a sensor system 108 with oneor more sub-systems for receiving information about the instruments ofteleoperational manipulator assembly 102. Such sub-systems may include aposition/location sensor system (e.g., an electromagnetic (EM) sensorsystem); a shape sensor system for determining the position,orientation, speed, velocity, pose, and/or shape of a distal end and/orof one or more segments along a flexible body that may make up medicalinstrument 104; and/or a visualization system for capturing images fromthe distal end of medical instrument 104.

Teleoperated medical system 100 also includes a display system 110 fordisplaying an image or representation of the surgical site and medicalinstrument 104 generated by sub-systems of sensor system 108. Displaysystem 110 and master assembly 106 may be oriented so physician O cancontrol medical instrument 104 and master assembly 106 with theperception of telepresence.

In some embodiments, medical instrument 104 may have a visualizationsystem (discussed in more detail below), which may include a viewingscope assembly that records a concurrent or real-time image of asurgical site and provides the image to the operator or physician Othrough one or more displays of medical system 100, such as one or moredisplays of display system 110. The concurrent image may be, forexample, a two or three dimensional image captured by an endoscopepositioned within the surgical site. In some embodiments, thevisualization system includes endoscopic components that may beintegrally or removably coupled to medical instrument 104. However insome embodiments, a separate endoscope, attached to a separatemanipulator assembly may be used with medical instrument 104 to imagethe surgical site. The visualization system may be implemented ashardware, firmware, software or a combination thereof which interactwith or are otherwise executed by one or more computer processors, whichmay include the processors of a control system 112.

Display system 110 may also display an image of the surgical site andmedical instruments captured by the visualization system. In someexamples, teleoperated medical system 100 may configure medicalinstrument 104 and controls of master assembly 106 such that therelative positions of the medical instruments are similar to therelative positions of the eyes and hands of physician O. In this mannerphysician O can manipulate medical instrument 104 and the hand controlas if viewing the workspace in substantially true presence. By truepresence, it is meant that the presentation of an image is a trueperspective image simulating the viewpoint of a physician that isphysically manipulating medical instrument 104.

In some examples, display system 110 may present images of a surgicalsite recorded pre-operatively or intra-operatively using image data fromimaging technology such as, computed tomography (CT), magnetic resonanceimaging (MRI), fluoroscopy, thermography, ultrasound, optical coherencetomography (OCT), thermal imaging, impedance imaging, laser imaging,nanotube X-ray imaging, and/or the like. The pre-operative orintra-operative image data may be presented as two-dimensional,three-dimensional, or four-dimensional (including e.g., time based orvelocity based information) images and/or as images from models createdfrom the pre-operative or intra-operative image data sets.

In some embodiments, often for purposes of imaged guided surgicalprocedures, display system 110 may display a virtual navigational imagein which the actual location of medical instrument 104 is registered(i.e., dynamically referenced) with the preoperative or concurrentimages/model. This may be done to present the physician O with a virtualimage of the internal surgical site from a viewpoint of medicalinstrument 104. In some examples, the viewpoint may be from a tip ofmedical instrument 104. An image of the tip of medical instrument 104and/or other graphical or alphanumeric indicators may be superimposed onthe virtual image to assist physician O controlling medical instrument104. In some examples, medical instrument 104 may not be visible in thevirtual image.

In some embodiments, display system 110 may display a virtualnavigational image in which the actual location of medical instrument104 is registered with preoperative or concurrent images to present thephysician O with a virtual image of medical instrument 104 within thesurgical site from an external viewpoint. An image of a portion ofmedical instrument 104 or other graphical or alphanumeric indicators maybe superimposed on the virtual image to assist physician in the controlof medical instrument 104. As described herein, visual representationsof data points may be rendered to display system 110. For example,measured data points, moved data points, registered data points, andother data points described herein may be displayed on display system110 in a visual representation. The data points may be visuallyrepresented in a user interface by a plurality of points or dots ondisplay system 110 or as a rendered model, such as a mesh or wire modelcreated based on the set of data points. In some examples, the datapoints may be color coded according to the data they represent. In someembodiments, a visual representation may be refreshed in display system110 after each processing operation has been implemented to alter datapoints.

Teleoperated medical system 100 may also include control system 112.Control system 112 includes at least one memory and at least onecomputer processor (not shown) for effecting control between medicalinstrument 104, master assembly 106, sensor system 108, and displaysystem 110. Control system 112 also includes programmed instructions(e.g., a non-transitory machine-readable medium storing theinstructions) to implement some or all of the methods described inaccordance with aspects disclosed herein, including instructions forproviding information to display system 110. While control system 112 isshown as a single block in the simplified schematic of FIG. 1 , thesystem may include two or more data processing circuits with one portionof the processing optionally being performed on or adjacent toteleoperational manipulator assembly 102, another portion of theprocessing being performed at master assembly 106, and/or the like. Theprocessors of control system 112 may execute instructions comprisinginstruction corresponding to processes disclosed herein and described inmore detail below. Any of a wide variety of centralized or distributeddata processing architectures may be employed. Similarly, the programmedinstructions may be implemented as a number of separate programs orsubroutines, or they may be integrated into a number of other aspects ofthe teleoperational systems described herein. In one embodiment, controlsystem 112 supports wireless communication protocols such as Bluetooth,IrDA, HomeRF, IEEE 802.11, DECT, and Wireless Telemetry.

In some embodiments, control system 112 may receive force and/or torquefeedback from medical instrument 104. Responsive to the feedback,control system 112 may transmit signals to master assembly 106. In someexamples, control system 112 may transmit signals instructing one ormore actuators of teleoperational manipulator assembly 102 to movemedical instrument 104. Medical instrument 104 may extend into aninternal surgical site within the body of patient P via openings in thebody of patient P. Any suitable conventional and/or specializedactuators may be used. In some examples, the one or more actuators maybe separate from, or integrated with, teleoperational manipulatorassembly 102. In some embodiments, the one or more actuators andteleoperational manipulator assembly 102 are provided as part of ateleoperational cart positioned adjacent to patient P and operatingtable T.

Control system 112 may optionally further include a virtualvisualization system to provide navigation assistance to physician Owhen controlling medical instrument 104 during an image-guided surgicalprocedure. Virtual navigation using the virtual visualization system maybe based upon reference to an acquired preoperative or intraoperativedataset of anatomic passageways. The virtual visualization systemprocesses images of the surgical site imaged using imaging technologysuch as computerized tomography (CT), magnetic resonance imaging (MRI),fluoroscopy, thermography, ultrasound, optical coherence tomography(OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-rayimaging, and/or the like. Software, which may be used in combinationwith manual inputs, is used to convert the recorded images intosegmented two dimensional or three dimensional composite representationof a partial or an entire anatomic organ or anatomic region. An imagedata set is associated with the composite representation. The compositerepresentation and the image data set describe the various locations andshapes of the passageways and their connectivity. The images used togenerate the composite representation may be recorded preoperatively orintra-operatively during a clinical procedure. In some embodiments, avirtual visualization system may use standard representations (i.e., notpatient specific) or hybrids of a standard representation and patientspecific data. The composite representation and any virtual imagesgenerated by the composite representation may represent the staticposture of a deformable anatomic region during one or more phases ofmotion (e.g., during an inspiration/expiration cycle of a lung).

During a virtual navigation procedure, sensor system 108 may be used tocompute an approximate location of medical instrument 104 with respectto the anatomy of patient P. The location can be used to produce bothmacro-level (external) tracking images of the anatomy of patient P andvirtual internal images of the anatomy of patient P. The system mayimplement one or more electromagnetic (EM) sensor, fiber optic sensors,and/or other sensors to register and display a medical implementtogether with preoperatively recorded surgical images., such as thosefrom a virtual visualization system, are known. For example U.S. patentapplication Ser. No. 13/107,562 (filed May 13, 2011) (disclosing“Medical System Providing Dynamic Registration of a Model of an AnatomicStructure for Image-Guided Surgery”) which is incorporated by referenceherein in its entirety, discloses one such system. Teleoperated medicalsystem 100 may further include optional operations and support systems(not shown) such as illumination systems, steering control systems,irrigation systems, and/or suction systems. In some embodiments,teleoperated medical system 100 may include more than oneteleoperational manipulator assembly and/or more than one masterassembly. The exact number of teleoperational manipulator assemblieswill depend on the surgical procedure and the space constraints withinthe operating room, among other factors. Master assembly 106 may becollocated or they may be positioned in separate locations. Multiplemaster assemblies allow more than one operator to control one or moreteleoperational manipulator assemblies in various combinations.

FIG. 2A is a simplified diagram of a medical instrument system 200according to some embodiments. In some embodiments, medical instrumentsystem 200 may be used as medical instrument 104 in an image-guidedmedical procedure performed with teleoperated medical system 100. Insome examples, medical instrument system 200 may be used fornon-teleoperational exploratory procedures or in procedures involvingtraditional manually operated medical instruments, such as endoscopy.Optionally medical instrument system 200 may be used to gather (i.e.,measure) a set of data points corresponding to locations within anatomicpassageways of a patient, such as patient P.

Medical instrument system 200 includes elongate device 202 coupled to adrive unit 204. Elongate device 202 includes a flexible body 216 havingproximal end 217 and distal end 218 (also called “tip portion 218”). Insome embodiments, flexible body 216 has an approximately 3 mm outerdiameter. Other flexible body outer diameters may be larger or smaller.

Medical instrument system 200 further includes a tracking system 230 fordetermining the position, orientation, speed, velocity, pose, and/orshape of flexible body 216 at distal end 218 and/or of one or moresegments 224 along flexible body 216 using one or more sensors and/orimaging devices as described in further detail below. The entire lengthof flexible body 216, between distal end 218 and proximal end 217, maybe effectively divided into segments 224. If medical instrument system200 is consistent with medical instrument 104 of a teleoperated medicalsystem 100, tracking system 230. Tracking system 230 may optionally beimplemented as hardware, firmware, software or a combination thereofwhich interact with or are otherwise executed by one or more computerprocessors, which may include the processors of control system 112 inFIG. 1 .

Tracking system 230 may optionally track distal end 218 and/or one ormore of the segments 224 using a shape sensor 222. Shape sensor 222 mayoptionally include an optical fiber aligned with flexible body 216(e.g., provided within an interior channel (not shown) or mountedexternally). In one embodiment, the optical fiber has a diameter ofapproximately 200 μm. In other embodiments, the dimensions may be largeror smaller. The optical fiber of shape sensor 222 forms a fiber opticbend sensor for determining the shape of flexible body 216. In onealternative, optical fibers including Fiber Bragg Gratings (FBGs) areused to provide strain measurements in structures in one or moredimensions. Various systems and methods for monitoring the shape andrelative position of an optical fiber in three dimensions are describedin U.S. patent application Ser. No. 11/180,389 (filed Jul. 13, 2005)(disclosing “Fiber optic position and shape sensing device and methodrelating thereto”); U.S. patent application Ser. No. 12/047,056 (filedon Jul. 16, 2004) (disclosing “Fiber-optic shape and relative positionsensing”); and U.S. Pat. No. 6,389,187 (filed on Jun. 17, 1998)(disclosing “Optical Fibre Bend Sensor”), which are all incorporated byreference herein in their entireties. Sensors in some embodiments mayemploy other suitable strain sensing techniques, such as Rayleighscattering, Raman scattering, Brillouin scattering, and Fluorescencescattering. In some embodiments, the shape of flexible body 216 may bedetermined using other techniques. For example, a history of the distalend pose of flexible body 216 can be used to reconstruct the shape offlexible body 216 over the interval of time. In some embodiments,tracking system 230 may optionally and/or additionally track distal end218 using a position sensor system 220. Position sensor system 220 maycomprise, or be a component of, an EM sensor system including one ormore conductive coils that may be subjected to an externally generatedelectromagnetic field. Each coil of an EM sensor system used toimplement position sensor system 220 then produces an induced electricalsignal having characteristics that depend on the position andorientation of the coil relative to the externally generatedelectromagnetic field. In some embodiments, position sensor system 220may be configured and positioned to measure six degrees of freedom,e.g., three position coordinates X, Y, Z and three orientation anglesindicating pitch, yaw, and roll of a base point or five degrees offreedom, e.g., three position coordinates X, Y, Z and two orientationangles indicating pitch and yaw of a base point. Further description ofa position sensor system is provided in U.S. Pat. No. 6,380,732 (filedAug. 11, 1999) (disclosing “Six-Degree of Freedom Tracking System Havinga Passive Transponder on the Object Being Tracked”), which isincorporated by reference herein in its entirety.

In some embodiments, tracking system 230 may alternately and/oradditionally rely on historical pose, position, or orientation datastored for a known point of an instrument system along a cycle ofalternating motion, such as breathing. This stored data may be used todevelop shape information about flexible body 216. In some examples, aseries of positional sensors (not shown), such as electromagnetic (EM)sensors similar to the sensors in position sensor system 220 may bepositioned along flexible body 216 and then used for shape sensing. Insome examples, a history of data from one or more of these sensors takenduring a procedure may be used to represent the shape of elongate device202, particularly if an anatomic passageway is generally static.

Flexible body 216 includes a channel 221 sized and shaped to receive amedical instrument 226. FIG. 2B is a simplified diagram of flexible body216 with medical instrument 226 extended according to some embodiments.In some embodiments, medical instrument 226 may be used for proceduressuch as surgery, biopsy, ablation, illumination, irrigation, or suction.Medical instrument 226 can be deployed through channel 221 of flexiblebody 216 and used at a target location within the anatomy. Medicalinstrument 226 may include, for example, image capture probes, biopsyinstruments, laser ablation fibers, and/or other surgical, diagnostic,or therapeutic tools. Medical tools may include end effectors having asingle working member such as a scalpel, a blunt blade, an opticalfiber, an electrode, and/or the like. Other end effectors may include,for example, forceps, graspers, scissors, clip appliers, and/or thelike. Other end effectors may further include electrically activated endeffectors such as electrosurgical electrodes, transducers, sensors,and/or the like. In various embodiments, medical instrument 226 is abiopsy instrument, which may be used to remove sample tissue or asampling of cells from a target anatomic location. Medical instrument226 may be used with an image capture probe also within flexible body216. In various embodiments, medical instrument 226 may be an imagecapture probe that includes a distal portion with a stereoscopic ormonoscopic camera at or near distal end 218 of flexible body 216 forcapturing images (including video images) that are processed by avisualization system 231 for display and/or provided to tracking system230 to support tracking of distal end 218 and/or one or more of thesegments 224. The image capture probe may include a cable coupled to thecamera for transmitting the captured image data. In some examples, theimage capture instrument may be a fiber-optic bundle, such as afiberscope, that couples to visualization system 231. The image captureinstrument may be single or multi-spectral, for example capturing imagedata in one or more of the visible, infrared, and/or ultravioletspectrums. Alternatively, medical instrument 226 may itself be the imagecapture probe. Medical instrument 226 may be advanced from the openingof channel 221 to perform the procedure and then retracted back into thechannel when the procedure is complete. Medical instrument 226 may beremoved from proximal end 217 of flexible body 216 or from anotheroptional instrument port (not shown) along flexible body 216.

Medical instrument 226 may additionally house cables, linkages, or otheractuation controls (not shown) that extend between its proximal anddistal ends to controllably the bend distal end of medical instrument226. Steerable instruments are described in detail in U.S. Pat. No.7,316,681 (filed on Oct. 4, 2005) (disclosing “Articulated SurgicalInstrument for Performing Minimally Invasive Surgery with EnhancedDexterity and Sensitivity”) and U.S. patent application Ser. No.12/286,644 (filed Sep. 30, 2008) (disclosing “Passive Preload andCapstan Drive for Surgical Instruments”), which are incorporated byreference herein in their entireties.

Flexible body 216 may also house cables, linkages, or other steeringcontrols (not shown) that extend between drive unit 204 and distal end218 to controllably bend distal end 218 as shown, for example, by brokendashed line depictions 219 of distal end 218. In some examples, at leastfour cables are used to provide independent “up-down” steering tocontrol a pitch of distal end 218 and “left-right” steering to control ayaw of distal end 281. Steerable catheters are described in detail inU.S. patent application Ser. No. 13/274,208 (filed Oct. 14, 2011)(disclosing “Catheter with Removable Vision Probe”), which isincorporated by reference herein in its entirety. In embodiments inwhich medical instrument system 200 is actuated by a teleoperationalassembly, drive unit 204 may include drive inputs that removably coupleto and receive power from drive elements, such as actuators, of theteleoperational assembly. In some embodiments, medical instrument system200 may include gripping features, manual actuators, or other componentsfor manually controlling the motion of medical instrument system 200.Elongate device 202 may be steerable or, alternatively, the system maybe non-steerable with no integrated mechanism for operator control ofthe bending of distal end 218. In some examples, one or more lumens,through which medical instruments can be deployed and used at a targetsurgical location, are defined in the walls of flexible body 216.

In some embodiments, medical instrument system 200 may include aflexible bronchial instrument, such as a bronchoscope or bronchialcatheter, for use in examination, diagnosis, biopsy, or treatment of alung. Medical instrument system 200 is also suited for navigation andtreatment of other tissues, via natural or surgically created connectedpassageways, in any of a variety of anatomic systems, including thecolon, the intestines, the kidneys and kidney calices, the brain, theheart, the circulatory system including vasculature, and/or the like.

The information from tracking system 230 may be sent to a navigationsystem 232 where it is combined with information from visualizationsystem 231 and/or the preoperatively obtained models to provide thephysician, clinician, or surgeon or other operator with real-timeposition information. In some examples, the real-time positioninformation may be displayed on display system 110 of FIG. 1 for use inthe control of medical instrument system 200. In some examples, controlsystem 116 of FIG. 1 may utilize the position information as feedbackfor positioning medical instrument system 200. Various systems for usingfiber optic sensors to register and display a surgical instrument withsurgical images are provided in U.S. patent application Ser. No.13/107,562, filed May 13, 2011, disclosing, “Medical System ProvidingDynamic Registration of a Model of an Anatomic Structure forImage-Guided Surgery,” which is incorporated by reference herein in itsentirety.

In some examples, medical instrument system 200 may be teleoperatedwithin medical system 100 of FIG. 1 . In some embodiments,teleoperational manipulator assembly 102 of FIG. 1 may be replaced bydirect operator control. In some examples, the direct operator controlmay include various handles and operator interfaces for hand-heldoperation of the instrument.

FIGS. 3A and 3B are simplified diagrams of side views of a patientcoordinate space including a medical instrument mounted on an insertionassembly according to some embodiments. As shown in FIGS. 3A and 3B, asurgical environment 300 includes a patient P is positioned on platform302. Patient P may be stationary within the surgical environment in thesense that gross patient movement is limited by sedation, restraint,and/or other means. Cyclic anatomic motion including respiration andcardiac motion of patient P may continue, unless patient is asked tohold his or her breath to temporarily suspend respiratory motion.Accordingly, in some embodiments, data may be gathered at a specific,phase in respiration, and tagged and identified with that phase. In someembodiments, the phase during which data is collected may be inferredfrom physiological information collected from patient P. Within surgicalenvironment 300, a point gathering instrument 304 is coupled to aninstrument carriage 306. In some embodiments, point gathering instrument304 may use EM sensors, shape-sensors, and/or other sensor modalities.Instrument carriage 306 is mounted to an insertion stage 308 fixedwithin surgical environment 300. Alternatively, insertion stage 308 maybe movable but have a known location (e.g., via a tracking sensor orother tracking device) within surgical environment 300. Instrumentcarriage 306 may be a component of a teleoperational manipulatorassembly (e.g., teleoperational manipulator assembly 102) that couplesto point gathering instrument 304 to control insertion motion (i.e.,motion along the A axis) and, optionally, motion of a distal end 318 ofan elongate device 310 in multiple directions including yaw, pitch, androll. Instrument carriage 306 or insertion stage 308 may includeactuators, such as servomotors, (not shown) that control motion ofinstrument carriage 306 along insertion stage 308.

Elongate device 310 is coupled to an instrument body 312. Instrumentbody 312 is coupled and fixed relative to instrument carriage 306. Insome embodiments, an optical fiber shape sensor 314 is fixed at aproximal point 316 on instrument body 312. In some embodiments, proximalpoint 316 of optical fiber shape sensor 314 may be movable along withinstrument body 312 but the location of proximal point 316 may be known(e.g., via a tracking sensor or other tracking device). Shape sensor 314measures a shape from proximal point 316 to another point such as distalend 318 of elongate device 310. Point gathering instrument 304 may besubstantially similar to medical instrument system 200.

A position measuring device 320 provides information about the positionof instrument body 312 as it moves on insertion stage 308 along aninsertion axis A. Position measuring device 320 may include resolvers,encoders, potentiometers, and/or other sensors that determine therotation and/or orientation of the actuators controlling the motion ofinstrument carriage 306 and consequently the motion of instrument body312. In some embodiments, insertion stage 308 is linear. In someembodiments, insertion stage 308 may be curved or have a combination ofcurved and linear sections.

FIG. 3A shows instrument body 312 and instrument carriage 306 in aretracted position along insertion stage 308. In this retractedposition, proximal point 316 is at a position L0 on axis A. In thisposition along insertion stage 308 an A component of the location ofproximal point 316 may be set to a zero and/or another reference valueto provide a base reference to describe the position of instrumentcarriage 306, and thus proximal point 316, on insertion stage 308. Withthis retracted position of instrument body 312 and instrument carriage306, distal end 318 of elongate device 310 may be positioned just insidean entry orifice of patient P. Also in this position, position measuringdevice 320 may be set to a zero and/or the another reference value(e.g., I=0). In FIG. 3B, instrument body 312 and instrument carriage 306have advanced along the linear track of insertion stage 308 and distalend 318 of elongate device 310 has advanced into patient P. In thisadvanced position, the proximal point 316 is at a position L1 on theaxis A. In some examples, encoder and/or other position data from one ormore actuators controlling movement of instrument carriage 306 alonginsertion stage 308 and/or one or more position sensors associated withinstrument carriage 306 and/or insertion stage 308 is used to determinethe position Lx of proximal point 316 relative to position L0. In someexamples, position LX may further be used as an indicator of thedistance or insertion depth to which distal end 318 of elongate device310 is inserted into the passageways of the anatomy of patient P.

FIG. 4 is a simplified diagram of a graphical user interface 400displayable on display system 110 according to some embodiments. In someembodiments consistent with FIGS. 1-3 , graphical user interface 400 maybe used to assist an operator, such as the physician, clinician, orsurgeon O, during the operation and/or control of a medical instrumentsystem, such as teleoperational manipulator assembly 100 and/or medicalinstrument system 200. Graphical user interface 400 displays informationin one or more windows 410-460 that are viewable to the operator.Although six concurrently viewable windows on a single screen aredepicted in FIG. 4 , it is to be understood that graphical userinterface 400 may display any suitable number of windows displayed onany suitable number of screens. In some examples, the number ofconcurrently viewable windows may be varied by opening and closingwindows, minimizing and maximizing windows, moving windows between aforeground and background of graphical user interface 400, switchingbetween screens, and/or otherwise fully or partially obscuring windowsfrom view. Similarly, the arrangement of windows 410-460—including theirsize, shape, orientation, ordering (in case of overlapping windows),and/or the like—may vary and/or may be user-configurable.

According to some embodiments, windows 410-460 may display image data,sensor data, indicators, control modes, and/or any combination thereof.In some examples, image data may include pre-operative orintra-operative image data. Image data may be presented astwo-dimensional, three-dimensional, or four-dimensional (including e.g.,time based or velocity based information) live images and/or as imagesof calculated models created from pre-operative or intra-operative imagedata sets. In some examples, images of calculated models may be derivedfrom sensor data, and may include models of instruments introduced intothe anatomy. In some examples, the calculated models may be created fromempirical data (in addition to or instead of image data) and/or may bebased on a predetermined geometry of instruments and/or human anatomy.In some examples, indicators may include graphical and/or alphanumericindicators. In some examples, controls may include buttons, text inputs,navigation panels, taskbars, icons, alerts, and/or the like. Accordingto some embodiments, graphical user interface 400 may include a settingswindow 450 that displays available control modes, current control mode,and/or a list of settings associated with the medical instrument system.

As depicted in FIG. 4 , one example of a graphical user interface 400includes a target guidance window 410 and virtual global view windows420 and 430 According to some embodiments, the data displayed in windows410-430 may include virtual images generated by a virtual visualizationsystem, such as virtual visualization system of control system 112.

Target guidance view window 410 displays a target location from aviewing angle corresponding to a distal end of the elongate device.According to some embodiments, target guidance view window 410 maydisplay guidance information designed to assist the operator in steeringthe elongate device to the target location from close range.

Virtual global view windows 420 and 430 display virtual image data fromviewing angles that provide a global view of patient P. In this manner,virtual global view window 430 simulates the field of view of anobserver, such as surgeon O. In some examples, the virtual image datamay display the real time position of the elongate device in the patientanatomy. In some examples, the respective viewing angles of virtualglobal view windows 470 and 480 may be selected manually and/orautomatically. According to some embodiments, the respective viewingangles of virtual global view windows 470 and 480 may be rotatedrelative to each other. In some examples, the viewing angles have afixed offset (e.g., a 90 degree offset to preserve orthogonality), suchthat rotating one of the perspectives causes the other to automaticallyrotate by a corresponding amount. In some examples, one or more of theviewing angles may be automatically selected to enhance the viewabilityof one or more bends in the elongate device. In some examples, one ofthe viewing angles may be automatically selected to match the view of afluoroscopic imaging device used to observe the procedure. An embodimentof virtual global view windows 420 and 430 is discussed in greaterdetail below with reference to FIG. 5 .

A camera view window 440 displays video image data captured by avisualization system, such as a visualization system of medicalinstrument 104. For example, the video image data may include videocaptured by an endoscope and/or a stereoscopic or monoscopic camera ator near a distal end of the medical instrument. One or more indicatorsand/or controls may be superimposed on and/or displayed alongside theimage data to assist the operator in controlling the medical instrument.

When windows 410-440 are displayed concurrently, the images displayed inwindows 410-440 advantageously allow the operator to concurrentlymonitor and/or visualize the vicinity of the distal end of the medicalinstrument (via target guidance window 410 and/or camera view window440) as well as the three-dimensional pose of the medical instrument(via virtual global view windows 420 and 430) in relation to patientanatomy.

According to some embodiments, one or more of the images displayed inwindows 410-440 may be augmented to display supplemental guidanceinformation to the operator. Additionally, and/or alternately, graphicaluser interface 400 may display supplemental guidance information to theoperator using an actuation information window 460. In some examples,the supplemental guidance information may be used to alert the operatorto problems and/or potential problems that arise during the operation ofthe medical instrument. The supplemental guidance information mayadditionally provide assistance to the operator in correcting, avoiding,and/or alleviating a detected problem.

According to some embodiments, the supplemental guidance information maybe associated with data from a tracking system, such as tracking system230. The data from the tracking system may indicate a position,orientation, speed, velocity, pose, and/or shape of the elongate deviceand/or portions thereof. The data from the tracking system may furtherindicate chemical, biological, mechanical, and/or thermal conditions ofthe elongate device and/or portions thereof. For example, as discussedabove with respect to FIG. 2A, the tracking system may include a shapesensor, such as shape sensor 222 and/or a fiber optic bend sensor, fordetermining the shape of an elongate device, such as elongate device202. Data from the shape sensor may be used to further calculatespecific conditions along the length of the elongate device (e.g. bendradius, buckling strain, activation force, temperature, and/or twist).Alternatively or additionally, the tracking system may include one ormore application-specific sensors, such as temperature, force, straingauge, electromagnetic, and/or pressure sensors, to measure conditionsalong the length of the elongate device. Such conditions could providecontinuous supplemental guidance information or alerts to the operatorwhen conditions reach a pre-determined threshold.

In one example, the supplemental guidance information may alert theoperator to excessive bending of the elongate device detected by thetracking system. Whether an excessive bending condition exists maydepend on the anatomy of the patient, such as patient P, the materialsand/or design of the elongate device, the materials and/or design of amedical tool inserted into the elongate device, and/or the like. Forexample, excessive bending may be problematic because it prevents amedical tool inserted into the elongate device from reaching a distalend of the elongate device and/or may cause a kink to form in theelongate device. In some examples, a medical tool and/or a discreteportion of the medical tool inserted into the elongate device may bestiffer than the flexible body of the elongate device. Thus, when theelongate device is in a configuration where one or more portions of theelongate device are excessively bent, it may be difficult and/orimpossible for the medical device to be delivered past the excessivelybent portion(s). In furtherance of such examples, the supplementalguidance information may assist the operator in detecting and correctingthe excessive bending condition prior to inserting the medical tool intothe patient. This prevents the operator from repeatedly inserting andremoving the medical tool from elongate device to manually discover by a“guess-and-check” method when one or more portions of the elongatedevice are excessively bent. According to some embodiments, thesupplemental guidance information may alert the operator to a variety ofother problems that arise during operation of the medical instrumentsystem, such as buckling, excessive strain, excessive twist, excessiveforce, out of range temperature, blockages in an anatomical passageway,anomalies detected by the tracking system (e.g., anomalies in thechemical, biological, mechanical, and/or thermal environment of theelongate device), and/or the like.

FIG. 5 is a simplified diagram of a window 500 for displaying virtualimage data augmented with supplemental guidance information according tosome embodiments. According to some embodiments consistent with FIGS.1-4 , window 500 may correspond to one or more of virtual global viewwindows 420 and 430. In this example, the image data in window 500depicts an elongate device 510, anatomical features 520, and a target530. According to some embodiments, anatomical features 520 may includeanatomical features of interest, such as anatomical passageways, bloodvessels, organs, and/or the like.

Target 530 identifies a point or region of the patient's anatomy wherean operator intends to guide elongate device 510. As depicted in FIG. 5, target 530 is depicted as a sphere. According to some embodiments,target 530 may be omitted. For example, anatomical features 520 mayprovide sufficient guidance to the operator to steer elongate device 510to a particular location even without target 530.

In some embodiments, image data that depicts elongate device 510,anatomical features 520, and target 530 may provide the operator withsufficient guidance information to steer elongate device 510 to aparticular location within a patient, such as patient P. However, inmany instances the operator may have difficulty detecting and correctingfor problems encountered by elongate device 510 during insertion. Forexample, in some images, the operator may have difficulty determiningthe bend radius of elongate device 510 (and/or portions thereof) withprecision. In some images, the operator may have difficultydistinguishing steerable portions of elongate device 510 fromnon-steerable portions. As a result, these images may be inadequate forcontrolling elongate device 510 in a manner that ensures that the bendradius of elongate device 510 does not exceed one or more predeterminedthreshold values. Similarly, even if the operator were to determine thatelongate device 510 was excessively bent, conventional images may notprovide sufficient guidance information for the operator to be able tocorrect the problem without resorting to inefficient methods such as“guess-and-check” to determine which direction to steer elongate device510 to remedy the problem.

To address these deficiencies, the image data displayed in window 500 isaugmented to display supplemental guidance information to the operator.Supplemental guidance information may be determined using a trackingsystem, such as a fiber optic bend sensor disposed along the length ofelongate device 510 as previously described. In one embodiment, thesupplemental guidance information is conveyed only when a pre-determinedthreshold is passed. In alternative embodiments, the supplementalguidance information may be conveyed to the operator continuously by wayof a color scheme 542, an alert icon 544, haptic/audio alerts 546,structural indicators 548, numerical values and/or the like.

In one example, color scheme 542 indicates the bend radius of elongatedevice 510 at different positions along elongate device 510. Colorscheme 542 can be used to display the measured bend radius by varyingthe color, texture, pattern, transparency, shade and/or another visualproperty of elongate device 510 as a function of position. Using colorscheme 542, different colors and/or shades may be assigned to ranges ofbend radius values (e.g., green may be assigned to a range that isconsidered straight and red may be assigned to a range that isconsidered bent, while yellow may be assigned to intermediate ranges).As depicted in FIG. 5 , a color scheme is adopted in which darkerportions of elongate device 510 correspond to a smaller bend radius.Such a color scheme may alert the operator to possible portions ofelongate device 510 that are excessively bent. For example, a region ofelongate device 510 may turn red when bent beyond a threshold value.According to some embodiments, the threshold value may correspond to abend radius at which a medical tool, such as a needle, can no longerfreely pass through elongate device 510. In some examples, the thresholdvalue may correspond to a minimum bend radius of elongate device 510,such as a radius at which elongate device 510 becomes susceptible toforming kinks. In some embodiments, there may be multiple thresholdvalues, with each threshold value triggering a different change in colorscheme 542, such as a transition to a darker hue of red to indicate thata more extreme threshold has been exceeded. Although transitions betweencolors are depicted as being abrupt, it is to be understood that colorscheme 542 may gradually transition between colors in some examples sothat a property of color scheme 542, such as hue, brightness, and/or thelike, is computed as a continuous function of the bend radius. In someexamples, color scheme 542 may be applied along the entire length ofelongate device 510. In some examples, color scheme 542 may be limitedto a distal portion of elongate device 510, as a proximal portion ofelongate device 510 may not be as susceptible as the distal portion tobecoming excessively bent.

According to some embodiments, alert icon 544 may appear in window 500when the bend radius exceeds one or more threshold values. As depictedin FIG. 5 , alert icon 544 is positioned in the vicinity of the problemarea to direct the attention of the operator to the region whereexcessive bending is identified. However, in some embodiments, alerticon 544 may appear at any position in window 500 such as adjacent thelocation along the length of elongate device 510 where the tight bendradius is measured. In some embodiments (not shown), alert icon 544 maydisplay alphanumeric values that quantify the bend radius, and/or mayvisually represent the magnitude of the bend radius using a gauge,meter, and/or the like. In some embodiments, the size, color, and/orother attribute of alert icon 544 may be proportional to the magnitudeof the bend radius. In some examples, alert icon 544 may include anarrow and/or other directional indicator to assist the operator inreducing the bend radius of elongate device 510.

According to some embodiments, haptic/audio alerts 546 and associatedgraphical icons may be used to alert the operator when the bend radiusexceeds one or more threshold values. For example, a haptic alert may betransmitted by vibrating a control device used by the operator tocontrol elongate device 510, such as a joystick, a trackball, and/or thelike. An audio alert may include an alarm signal and/or a voiceover thatstates the type of problem encountered. Haptic/audio alerts 546 may helpto draw the attention of the operator to window 500 when the bend radiusexceeds one or more threshold values even if the operator is lookingelsewhere.

According to some embodiments, the supplemental guidance information mayinclude structural indicators 548. In general, structural indicators mayindicate structural components of elongate device 510, such as a distalend of elongate device 510 and/or different segments of elongate device510, as a visual aid to the operator. As depicted FIG. 5 , structuralindicators 548 include steerable range indicators that visually bookenda portion of elongate device 510, if any, that is steerable. Infurtherance of such embodiments, structural indicators 548 may includelines and/or markers indicating positions along elongate device 510 atwhich a transition between steerable and non-steerable portions ofelongate device 510 occurs. In some examples, structural indicators 548may include arrows, crosshairs, and/or any other suitable indicator orset of indicators. In some embodiments, elongate device 510 may bedepicted using color scheme 542 within the steerable range and using anormal color scheme (e.g. gray) outside of the steerable range. In someexamples, a designated color may be used for structural indicators 548,such as blue.

According to some embodiments, various modifications may be made towindow 500 to improve the clarity and/or prominence with which thesupplemental guidance information is displayed. According to someembodiments, a viewing angle of window 500 may be selected to highlightthe portion of elongate device 510 with the tightest bend radius. Forexample, the viewing angle may be dynamically selected to be orthogonalto the plane of the tightest bend radius. In some examples, anatomicalfeatures 520 and/or target 530 may not be displayed in window 500 so asnot to draw attention from the supplemental guidance informationincluded in window 500.

FIG. 6 is a simplified diagram of a window 600 for displaying actuationinformation including supplemental guidance information according tosome embodiments. According to some embodiments consistent with FIGS.1-4 , window 600 may correspond to actuation information window 460.Like the augmented images displayed in window 500, window 600 is used todisplay supplemental guidance information associated with a bend radiusmeasured using a tracking system, such as a fiber optic bend sensordisposed along the length of an elongate device. According to someembodiments, window 600 may display one or more graphical indicatorsincluding an actuation information icon 610.

In general, actuation information icon 610 provides guidance informationthat assists an operator in correcting a problem encountered whilecontrolling the elongate device, such as excessive bending of theelongate device. In some examples, actuation information icon 610 mayinclude an alphanumeric indicator 612 that displays a minimum bendradius of the elongate device (i.e., the smallest bend radius along thelength of the elongate device) to alert the operator to an excessivebending condition. In some examples, alphanumeric indicator 612 maydisplay a numeric value which continually updates as the tightest bendradius changes but switches to an alpha value (e.g. YES or PASS) whenthe tightest bend radius equals a value that has been pre-determined tosafely allow the passage of a medical tool. In some examples, the valuemay be an alpha value that displays either a PASS or FAIL, YES or NO,and/or another binary indicator of a large enough bend radius down thelength of the elongate device to allow for the passage of a medicaltool.

In some examples, actuation information icon 610 may include adirectional indicator 614, such as an arrow, that indicates whichdirection the operator should steer the elongate device to alleviate theexcessive bending condition. In some examples, the color, size, texture,and/or other attributes of actuation information icon 610, alphanumericindicator 612, and/or directional indicator 614 may be dynamic so as toconvey supplemental guidance information to the operator. For example,different colors may correspond to different ranges of band radius(e.g., red corresponds to a bend radius of 1-10, green corresponds to abend radius over 50, and yellow— and/or a gradually shifting shade ofcolor from red to orange to yellow to green—corresponds to a bend radiusof 11-49). In some examples, the color scheme used to determine thecolor of actuation information icon 610 may match color scheme 542 ofwindow 500. The color scheme may be applied to action information icon610 or portions of thereof such as directional indicator 614 and/ornumerical indicator 612. In some examples, one or more of actuationinformation icon 610, alphanumeric indicator 612, and/or directionalindicator 614 may disappear when excessive bending is not detected.

In the illustrative example depicted in FIG. 6 , the minimum bend radiusof the elongate device is 11 mm, as depicted by alphanumeric indicator612. The smaller the number, the tighter the bend radius. In order toincrease the minimum bend radius, the operator is instructed to navigatea control device, such as a joystick, a trackball, and/or the like, downand to the left, as depicted by the arrow of directional indicator 614.For example, in one or more embodiments, actuation information icon 610may depict a top view of a trackball used by the operator to control thebend of a steerable portion of the elongate device. In furtherance ofsuch embodiments, directional indicator 614 may indicate the directionthe trackball should be rolled to straighten the steerable portion ofthe elongate device.

While the examples in FIGS. 4-6 have been largely described with respectto display of bend radius, it should be understood that the display inany of the previously described examples can similarly representconditions such as but not limited to buckling strain, activation force,temperature, and/or twist. As with bend, buckling, strain, activationforce, temperature, and/or twist may be displayed as varying colors,textures, patterns, transparencies, shades, alpha-numeric indicators,haptic/audible/visual alerts and/or visual properties along the lengthof the elongate device or at any location in any of the previouslydescribed display windows.

FIG. 7 is a simplified diagram of a method 700 of displayingsupplemental guidance information during an image-guided surgicalprocedure according to some embodiments. In some examples, method 700may be used to display supplemental guidance information on a graphicaluser interface, such as graphical user interface 400, and/or in a windowof a graphical user interface, such as windows 500 and/or 600. Method700 is illustrated in FIG. 7 as a set of operations or processes710-740. Not all of the illustrated processes 710-740 may be performedin all embodiments of method 700. Additionally, one or more processesthat are not expressly illustrated in FIG. 7 may be included before,after, in between, or as part of the processes 710-740. In someembodiments, one or more of the processes 710-740 of method 700 may beimplemented, at least in part, in the form of executable code stored onnon-transitory, tangible, computer readable media that when run by oneor more processors (e.g., the processors of control system 112) maycause the one or more processors to perform one or more of the processes710-740.

At a process 710, tracking data is received from a tracking systemassociated with an elongate device, such as elongate device 202.According to some embodiments, the tracking data may include informationassociated with the position, orientation, speed, velocity, environment(e.g., chemical, thermal, and/or biological environment), temperature,force, pose, and/or shape of a flexible body of the elongate device. Insome examples, the tracking data may include data collected from aplurality of points and/or segments along the length of the flexiblebody. In some examples, the tracking system may include a shape sensor,such as a fiber optic bend sensor disposed along the length of theflexible body. Consistent with such embodiments, the tracking data mayinclude a bend radius of the flexible body at various positions alongthe flexible body and/or sufficient information from which to determinethe bend radius. In additional embodiments, the tracking data may beused to calculate alternative conditions of the flexible body such asbuckling, strain, activation force, temperature, or twist.

At a process 720, supplemental guidance information is determined basedon the received tracking data. In some examples, the supplementalguidance information may be determined by comparing the probe data toone or more predetermined thresholds. For example, when the probe dataincludes a bend radius of the flexible body at various positions alongthe flexible body, the supplemental guidance may be determined bycomparing the bend radius to a minimum allowable bend radius of theflexible body. In some examples, the minimum allowable bend radius maybe selected to ensure the unimpeded passage of one or more medicaldevices through the flexible body. In some examples, the minimumallowable bend radius may be selected to prevent kinks or other damageto the flexible body.

In some examples, the minimum bend radius may be 10 mm or less. In someexamples, one or more threshold values may be configurable (e.g., set bythe operator) and/or may vary based on the type of surgical procedure,the model of the elongate device, the physical characteristics of thepatient, the types of medical devices being inserted into the flexiblebody, and/or the like. According to some embodiments, the supplementalguidance information may additionally, and/or alternately, includecorrective guidance information. For example, when the bend radius ofthe flexible body is smaller than a particular threshold, determiningthe corrective guidance information may include determining a directionin which to steer the flexible body in order to alleviate the excessivebending condition.

At a process 730, one or more images are augmented to include thesupplemental guidance information. According to some embodiments, theone or more images may correspond to virtual global images generated bya virtual visualization system, such as virtual visualization system ofcontrol system 112. Consistent with such embodiments, supplementalguidance information may be input into the virtual visualization systemto generate virtual images augmented using the supplemental guidanceinformation. Alternately or additionally, virtual images may be receivedfrom the virtual visualization system, and the supplemental guidanceinformation may be overlaid on the received virtual images. In someembodiments, the one or more images may correspond to video image datacaptured by a visualization system, such as a visualization system ofmedical instrument 104. Consistent with such embodiments, supplementalguidance information may be overlaid on the video image data. Asdiscussed previously with respect to FIG. 5 , the images may beaugmented by including a color scheme, alert icon, haptic/audioindicator, structural indicators, and/or the like. In some examples,augmenting the one or more images may include applying a color scheme tothe flexible body such that portions of the flexible body with a smallerbend radius are colored or shaded differently than portions of theflexible body with a larger bend radius. Similarly, augmenting the oneor more images may include applying structural indicators to theflexible body to demarcate one or more portions of the flexible bodythat are steerable. In some examples, the supplemental guidanceinformation may be displayed as actuation information, such as arrowsindicating a direction in which to steer the flexible body of theelongate device in order to reduce the bend.

At a process 740, the one or more augmented images are displayed on thegraphical user interface. In some examples, the one or more augmentedimages may be displayed in a window concurrently with a supplementalguidance window, such as supplemental guidance window 600. In someexamples, the one or more augmented images may be displayed concurrentlywith an audio and/or haptic alert to indicate whether a threshold valueis exceeded, such as when excessive bending of the flexible body isdetected. In some examples, the one or more augmented images may bedisplayed concurrently with an actuation information icon. For example,the actuation information icon may include a directional indicator todisplay the direction to steer the flexible body to increase the bendradius.

FIG. 8 is a screenshot 800 of a graphical user interface displayingsupplemental guidance information for use in an image-guided surgicalprocedure according to some embodiments. Like graphical user interface400, screenshot 800 depicts six concurrently viewable frames or windows810-860. The upper three frames 810-830 depict virtual imagescorresponding to a target guidance view and two virtual global views ofan elongate device inserted into an anatomical passageway. Orientationindicators 870 are displayed in the lower right corner of each of thevirtual images to indicate the viewing angle of each image relative to apatient. Virtual global views 820 and 830 illustrate a graphicalrepresentation of a 3D anatomical model 822 of a patient's lung and amodel 824 of an elongate device as it moves through the branches of thelung. In some examples, 3D anatomical model 822 may be generated frompre-operative CT scans and then registered to the patient's anatomy.Model 824 is also registered to the patient's anatomy, and therepresentation of the elongate device within model 824 is updated inreal-time based on tracking data previously described. Additionally, auser pre-selected target 880 is displayed in frames 810-830.

Like the image displayed in window 500, the virtual global images 820and 830 are augmented to display supplemental guidance information. Morespecifically, model 824 of the elongate device is colored according to acolor scheme that indicates the bend radius of the elongate device ateach point along its length. A red portion of model 824 indicates thatthe bend radius in that portion is within a critical range (e.g., belowa predetermined threshold value) where there is a risk of blocking thepassage of an instrument through the elongate device and/or strainingthe elongate, while green is assigned to portions of the elongate devicethat are deemed to be outside the critical range and not problematic interms of passage of an instrument and/or strain on the elongate device.In this particular example, red corresponds to a bend radius of 1-10,green corresponds to a bend radius over 50, and a gradually shiftingshade of color from red to orange to yellow to green corresponds to abend radius of 11-49. In addition, the blue lines on model 824 demarcatea portion of the elongate device that is steerable. Portions of elongatedevice in between the blue line are steerable; other portions are notsteerable.

The lower three frames or windows depicted in screenshot 800 include anendoscopic camera window 840, a control mode window 850, and anactuation information window 860. Like actuation information window 600,actuation information window 860 includes an actuation information icon865 that depicts a top view of a trackball used by the operator.Actuation information icon 865 alphanumerically indicates the minimumbend radius of the elongate device and graphically indicates thedirection in which to steer the elongate device (i.e., the direction theoperator should roll the trackball) to increase the minimum bend radiusand mitigate the excessive bending. The color of actuation informationicon 865 is dynamic and matches the color scheme of model 824: redcorresponds to a minimum bend radius of 1-10, green corresponds to aminimum bend radius over 50, and a gradually shifting shade of colorfrom red to orange to yellow to green corresponds to a minimum bendradius of 11-49.

Based on the color scheme of model 824, the color scheme of actuationinformation icon 865, and/or the alphanumeric bend radius indicator ofactuation information icon 865, the operator may observe an excessivebending condition of the elongate device. Moreover, the operator mayobserve an abrupt change in the bend of the elongate device at aposition where the expected shape of the elongate device is straight,such as at a position within or near a straight branch of an anatomicalpassageway. Based on these observations, the operator may reposition theelongate device manually and/or robotically using a trackball,particularly when the operator believes the displayed informationindicates a minor and/or fixable problem. Alternately or additionally,the operator may withdraw the elongate device partially and attempt adifferent approach to the target, and/or entirely withdraw the elongatedevice from the patient, particularly when the operator believes thedisplayed information indicates a severe and/or dangerous failure. Insome examples, this process may allow the operator to remove theelongate device from the patient before inserting a medical tool intothe elongate device, thereby preventing the failure from beingexacerbated by inserting the medical tool into a region where anundetected failure exists. In one or more embodiments, one or morethreshold values for bend radius may be used to assist the operator indetermining the severity of the excessive bending condition and theappropriate mitigating actions to take.

FIG. 9 is a simplified diagram of a bend indicator 900 according to someembodiments. According to some embodiments consistent with FIGS. 1-8 ,bend indicator 900 may be displayed in actuation information window 460.In some examples, bend indicator 900 may be displayed alongside anactuation information icon, such as actuation information icon 610,and/or as an alternative to the actuation information icon. However, itis to be understood that bend indicator 900 may be displayed in contextsother than graphical user interface 400, including as a standalone viewand/or in conjunction with views other than those depicted in graphicaluser interface 400. In some examples, bend indicator 900 may appear whena tight bend radius is detected in the catheter (e.g., when the bendradius is below a predetermined threshold) and may be hidden otherwise.Alternatively, selected portions of bend indicator 900 may be hiddenwhen no tight bend is present, e.g. numerical bend radius 910.

Bend indicator 900 provides a schematic bend representation 910 of thecatheter. When the distal tip of the catheter is bent, a bend line 925appears which indicates the direction the catheter distal end isbending. For example, as depicted in FIG. 9 , the bend line 925 appearson the upper right of a ring 915, indicating that the catheter is bentto the right. Thus to straighten the catheter, the catheter may besteered to the lower left to reduce the bend. In some examples, when thecatheter distal end is straight, bend line 925 may be hidden.

In some examples, schematic bend representation 910 may include arendering of a distal end of the catheter from the perspective oflooking backwards up the catheter tube through a distal tip of thecatheter (towards a proximal portion of the catheter from the distaltip). Consistent with such examples, ring 915 may be interpreted ascorresponding to the distal tip of the catheter. When the catheter isbent, portions of the catheter become visible behind the distal tip(i.e., ring 915). Consequently, bend line 925 may correspond to theportions of the distal end of the catheter that are visible behind thedistal tip (i.e., ring 915) due to the bending of the catheter.

In alternative examples, schematic bend representation 910 may include arendering of the distal end of the catheter from the perspective oflooking forward down the catheter tube towards the distal tip from aproximal position along the catheter. Consistent with such examples,ring 915 may be interpreted as corresponding to a cross-sectional cut ofthe catheter at the proximal position. When the catheter is bent,portions of the distal end become visible behind the cross-sectional cut(i.e., ring 915). Consequently, bend line 925 may correspond to theportions of the catheter that are visible behind the cross-sectional cut(i.e., ring 915) due to the bending of the catheter.

In some examples, bend indicator 900 may display a visual and/oralphanumeric representation of the minimum bend radius or the smallestbend radius detected along the catheter. When the minimum bend radiusdrops below a threshold value, bend indicator 900 may alert theclinician that the predetermined threshold has been breached bydisplaying an alphanumeric value and/or may otherwise changing inappearance. In some embodiments, the threshold value may be determinedbased on whether a tool can be passed through the catheter. In someembodiments, the threshold value may be determined based on the radiusat which buckling and/or damage to the catheter may occur. The thresholdvalue may be manually selected, automatically determined, determinedbased on the type of catheter and/or tool, and/or set using a generalrule of thumb. As depicted in FIG. 9 , when the minimum detected bendradius is below the threshold value, bend indicator 900 includes anumber 920 indicating the real-time value of the minimum bend radius,and portions of bend indicator 900 turn a different color, such as redas shown in FIG. 9 .

In some embodiments, the location of the red colored portions mayreflect the magnitude of the force applied by one of the motor to acatheter pull wire in that section of the catheter. For example, in FIG.9 the pull wire on the top left is being pulled harder, as indicated bythe red colored wedge appearing in schematic bend representation 910. Insome examples, bend indicator 900 may include an outer ring 930 thatdynamically changes color based whether the minimum bend radius isapproaching or exceeds the threshold value. In some examples, dynamicchanges could be represented by changes in appearance of portions of thebend indicator 900 in transparency, texture, line width, and/or coloretc.

Some examples of control units, such as control unit 130 may includenon-transient, tangible, machine readable media that include executablecode that when run by one or more processors (e.g., processor 140) maycause the one or more processors to provide the graphical user interface400 or perform the processes of method 700. Some common forms of machinereadable media that may provide the graphical user interface 400 orinclude the processes of method 700 are, for example, floppy disk,flexible disk, hard disk, magnetic tape, any other magnetic medium,CD-ROM, any other optical medium, punch cards, paper tape, any otherphysical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM,any other memory chip or cartridge, and/or any other medium from which aprocessor or computer is adapted to read.

ADDITIONAL EXAMPLES

A. A non-transitory machine-readable medium comprising a plurality ofmachine-readable instructions which when executed by one or moreprocessors associated with a medical device are adapted to cause the oneor more processors to perform a method comprising:

-   -   receiving tracking data associated with an elongate device;    -   calculating at least one condition along a length of the        elongate device based on the tracking data;    -   determining supplemental guidance information based on the at        least one condition;    -   augmenting one or more images including a graphical        representation of the elongate device with the supplemental        guidance information to produce one or more augmented images;        and    -   displaying the one or more augmented images on a display device        at a surgeon console.

B. The non-transitory machine-readable medium of example A, wherein thetracking data is received from a tracking system includes a shapesensor.

C. The non-transitory machine-readable medium of example B, wherein theshape sensor includes a fiber optic bend sensor.

D. The non-transitory machine-readable medium of any one of examplesA-C, wherein the at least one condition further comprises at least oneof a bend radius, buckling condition, strain, activation force,temperature, and twist of a flexible body of the elongate device.

E. The non-transitory machine-readable medium of example D, whereindetermining the supplemental guidance information includes determiningwhether the bend radius is less than a minimum allowable bend radius.

F. The non-transitory machine-readable medium of example E, wherein theminimum allowable bend radius is selected to allow passage of a medicaldevice through the elongate device.

G. The non-transitory machine-readable medium of example E, wherein theminimum allowable bend radius is selected to prevent a kink or damage tothe elongate device.

H. The non-transitory machine-readable medium of example D, whereinaugmenting the one or more images includes applying a color scheme tothe graphical representation of the elongate device, wherein portions ofthe graphical representation of the elongate device corresponding to asmaller bend radius are shaded differently than portions of thegraphical representation of the elongate device corresponding to alarger bend radius.

I. The non-transitory machine-readable medium of any one of examplesA-H, wherein the one or more images include structural indicators, thestructural indicators demarcating one or more portions of the elongatedevice that are steerable.

J. The non-transitory machine-readable medium of any one of examplesD-I, wherein the supplemental guidance information includes correctiveguidance information.

K. The non-transitory machine-readable medium of example J, wherein thecorrective guidance information includes a direction to steer theelongate device to increase the bend radius.

L. The non-transitory machine-readable medium of example K, wherein themethod further comprises displaying actuation information on the displaydevice, the actuation information including a directional indicator todisplay the direction to steer the flexible body to increase the bendradius.

M. The non-transitory machine-readable medium of any one of examplesA-L, wherein the one or more images include one or more virtual images.

N. The non-transitory machine-readable medium of any one of examplesA-L, wherein the one or more images include one or more video images.

O. The non-transitory machine-readable medium of example A, wherein themethod performed by the one or more processors further comprisesidentifying at least one of a position, an orientation, a speed, avelocity, an chemical environment, a thermal environment, a biologicalenvironment, a pose, and a shape of the elongate device from thetracking data.

P. The non-transitory machine-readable medium of claim O, wherein the atleast one of the position, the orientation, the speed, the velocity, thechemical environment, the thermal environment, the biologicalenvironment, the pose, and the shape is measured at a plurality ofpositions along the length of the elongate device.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

1-32. (canceled)
 33. A method for displaying guidance information, themethod comprising: receiving, by one or more hardware processors, datafrom a tracking system associated with an elongate device comprising aflexible body; determining, by the one or more hardware processors andbased on the received data, a buckling condition along a length of theflexible body; determining, by the one or more hardware processors andbased on the buckling condition, supplemental guidance information;augmenting, by the one or more hardware processors, one or more imageswith the supplemental guidance information to produce one or moreaugmented images; and displaying the one or more augmented images on adisplay device at a surgeon console.
 34. The method of claim 33, whereindetermining the buckling condition comprises comparing a bend radiusalong the length of the flexible body to a predetermined threshold valueassociated with buckling.
 35. The method of claim 33, wherein thetracking system includes a shape sensor.
 36. The method of claim 35,wherein the shape sensor includes a fiber optic bend sensor.
 37. Themethod of claim 33, wherein augmenting the one or more images includesapplying a color scheme to a graphical representation of the flexiblebody, wherein at least one portion of the graphical representation ofthe flexible body subject to the buckling condition is shadeddifferently than other portions of the graphical representation of theflexible body.
 38. The method of claim 33, wherein the one or moreimages include structural indicators, the structural indicatorsdemarcating one or more portions of the flexible body that aresteerable.
 39. The method of claim 33, wherein the supplemental guidanceinformation includes corrective guidance information.
 40. The method ofclaim 33, further comprising displaying actuation information on thedisplay device, the actuation information including a directionalindicator to display a direction to steer the flexible body to reducethe buckling condition.
 41. The method of claim 33, wherein the one ormore images include virtual images.
 42. The method of claim 33, whereindata from the tracking system is further associated with at least one ofa position, an orientation, a speed, a velocity, a chemical environment,a thermal environment, a biological environment, a pose, or a shape ofthe elongate device.
 43. A medical system comprising: an elongate deviceincluding a flexible body; a tracking system disposed along at least aportion of the flexible body; a display device; and one or moreprocessors coupled to the tracking system; wherein the one or moreprocessors are configured to: receive data from the tracking system;based on the received data, determine a buckling condition along alength of the flexible body; based on the buckling condition, determinesupplemental guidance information; using the supplemental guidanceinformation, augment one or more images to produce one or more augmentedimages; and display the one or more augmented images on the displaydevice.
 44. The medical system of claim 43, wherein the tracking systemincludes a shape sensor.
 45. The medical system of claim 44, wherein theshape sensor includes a fiber optic bend sensor.
 46. The medical systemof claim 43, wherein the one or more processors are configured toaugment the one or more images by applying a color scheme to a graphicalrepresentation of the flexible body, wherein at least one portion of thegraphical representation of the flexible body corresponding to thebuckling condition is shaded differently than other portions of thegraphical representation of the flexible body.
 47. The medical system ofclaim 43, wherein the one or more images include structural indicators,the structural indicators demarcating one or more portions of theflexible body that are steerable.
 48. The medical system of claim 43,wherein the supplemental guidance information includes correctiveguidance information.
 49. The medical system of claim 48, wherein thecorrective guidance information includes a direction to steer theflexible body to reduce the buckling condition and wherein the one ormore processors are further configured to display actuation informationon the display device, the actuation information including a directionalindicator to display the direction to steer the flexible body.
 50. Themedical system of claim 43, wherein the one or more images include oneor more virtual images.
 51. The medical system of claim 43, wherein theone or more images include one or more video images.
 52. The medicalsystem of claim 43, wherein the one or more processors are furtherconfigured to identify at least one of a position, an orientation, aspeed, a velocity, a chemical environment, a thermal environment, abiological environment, a pose, or a shape of the elongate device fromthe data received from the tracking system.